Aiding Expert Judges in Open-Innovation Challenges

A critical stage in open-innovation challenges is the process of screening and evaluating submissions, and selecting the best ones. It requires high-levels of expertise, especially where challenges and solutions are multi-faceted; when multiple evaluation criteria are used; and when some evaluation criteria lack objective measures. The scarcity of available expertise required for evaluation can create a bottleneck, which is exacerbated as the number of submissions grows. A notable case occurred when BP crowdsourced ideas for overcoming the DeepWater Horizon oil spill: it took a large multidisciplinary expert team several months to evaluate more than 120,000 submissions [BP, 2010]. Though this case is extreme, it exemplifies a challenge common to many open-innovation systems. While computational evaluation will not replace expert judges in such complex scenarios anytime soon, we demonstrate how reliable predictive models can be built and employed to separate the wheat from the chaff, potentially leading to significant reduction of expert review labor. Our approach is unique in that it considers not only properties of the submission itself, but also explicit and implicit traces of human activity in relation to submissions. We developed models based on data from 18 ideation contests run within the Climate CoLab, a large-scale citizen science framework, and were able to accurately predict expert decisions about the submissions, in this set, as well as in another set of proposals submitted to 18 additional CoLab contests which were reviewed by other judges. Our approach, and many of our metrics, can be adapted to other settings.